fix!: exclude downtime from crawler_runtime after a resume by Pijukatel · Pull Request #2204 · apify/crawlee-python

Pijukatel · 2026-08-31T13:49:01Z

After an Actor migration or resurrection, the first periodic statistics log reported crawler_runtime including the downtime between the runs (e.g. 38.01s for a ~0.5 s run); later logs corrected themselves. Now the first log reports exactly the runtime accumulated so far (0 for a fresh run).
Also covers two rarer edge cases: a state persisted mid-run and never finished cleanly baked the downtime into the runtime permanently, and repeated resurrections kept only the last run's segment of the accumulated runtime.
StatisticsState.accumulated_runtime now stores the runtime of the previous runs; the read-only crawler_runtime property adds the run in progress, and every persist checkpoints the live total.
Breaking (v2): crawler_runtime_for_serialization is removed, and crawlerRuntimeMillis is persisted as a number of milliseconds instead of an ISO 8601 duration string. Documented in docs/upgrading/upgrading_to_v2.md.
Runtime tracking matches Crawlee JS (accumulation across resumes, checkpoint on every persist, downtime excluded).
Related to Remove deprecated StatisticsState.crawler_runtime setter #1567.

codecov · 2026-08-31T13:52:33Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (v2@51efb37). Learn more about missing BASE report.

Additional details and impacted files

@@          Coverage Diff          @@
##             v2    #2204   +/-   ##
=====================================
  Coverage      ?   94.18%           
=====================================
  Files         ?      187           
  Lines         ?    13823           
  Branches      ?        0           
=====================================
  Hits          ?    13019           
  Misses        ?      804           
  Partials      ?        0

Flag	Coverage Δ
unit	`94.18% <100.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Pijukatel · 2026-09-03T07:22:08Z

Waiting for V2 as this might be breaking change and will inlcude removal of deprecated fields

…r resurrection The `crawler_runtime` statistic no longer counts the downtime between the runs of a resumed crawler, and it accumulates correctly across any number of persist/resume cycles: - The first periodic statistics log reports exactly the runtime accumulated so far: zero for a fresh start, the previously persisted total for a resumed run. - The accumulated runtime is stored in the new `accumulated_runtime` field, persisted under `crawlerRuntimeMillis`. Every persist writes the live total, so a state persisted mid-run (migration, abort) carries the runtime up to that moment and a resumed run continues from the last checkpoint. - The timestamps of the previous run are cleared before the new run starts being measured, so they cannot leak the downtime into the runtime, and a backwards clock jump during a run cannot push the runtime below the accumulated total. BREAKING CHANGE: `StatisticsState.crawler_runtime_for_serialization` is removed; the `crawler_runtime` property is backed by the new `accumulated_runtime` field. `crawlerRuntimeMillis` is serialized as a number of milliseconds instead of an ISO 8601 duration string. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy

apify-factory

Looks good, just one note.

Outside the diff:

src/crawlee/statistics/_models.py: Note: crawlerRuntimeMillis is serialized as the total (including the last run's segment) but validated back into accumulated_runtime alongside the unchanged crawlerLastStartTimestamp/crawlerFinishedAt, so StatisticsState.model_validate(state.model_dump(by_alias=True)).crawler_runtime counts the last run twice. Only Statistics.__aenter__ clears the timestamps, so any consumer that parses a persisted statistics record with StatisticsState gets an inflated runtime.

🏭 Factory review — ℹ️ not auto-approved

This change was reviewed but cannot be auto-approved: a gate failed, so the review cannot end in an approval.

check	result
Static gates	❌ 13/16 — `base-branch`, `max-lines`, `pr-title`
Review findings	❌ not clean
Reviewer escalation	❌ Changes the persisted statistics format (`crawlerRuntimeMillis` type and meaning) and removes a public model field, a breaking public contract that downstream consumers like the Apify SDK may read.
Juror 1 — `claude-sonnet-5-5`, adversarial	⏭️ skipped — a gate failed

Why it was not auto-approved

Failed gates:

base-branch: base is v2, required master
max-lines: 217 changed lines (+194/-23), limit 200
pr-title: breaking changes are never auto-approved

The comments above are the review itself; this section only explains the approval decision. Push a change and request @apify-factory again to re-run it, or ask a human to review.

_{Reviewed fc873682f · workflow run}

Pijukatel · 2026-10-21T07:56:08Z

src/crawlee/statistics/_models.py: Note: crawlerRuntimeMillis is serialized as the total (including the last run's segment) but validated back into accumulated_runtime alongside the unchanged crawlerLastStartTimestamp/crawlerFinishedAt, so StatisticsState.model_validate(state.model_dump(by_alias=True)).crawler_runtime counts the last run twice. Only Statistics.aenter clears the timestamps, so any consumer that parses a persisted statistics record with StatisticsState gets an inflated runtime.

That is not how the StatisticsState should be used, and no Crawlee or SDK code is doing that.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy

vdusek · 2026-10-27T17:43:26Z

The title is 84 characters long. Could we use something more concise?

E.g.:

"fix!: exclude downtime from crawler_runtime after a resume"
- resume should cover both migration and ressurrection
"fix!: report correct crawler runtime after Actor migration or resurrection"
- this points more at the user visible sympton rather than the internal field

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

…ed manually.

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

vdusek

Last 3 nits, otherwise LGTM

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

github-actions Bot assigned Pijukatel Aug 31, 2026

github-actions Bot added this to the 148th sprint - Tooling team milestone Aug 31, 2026

github-actions Bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Aug 31, 2026

Pijukatel modified the milestones: 148th sprint - Tooling team, 2.0 Sep 3, 2026

Pijukatel changed the title ~~fix: do not count downtime towards crawler_runtime after migration or resurrection~~ fix!: do not count downtime towards crawler_runtime after migration or resurrection Sep 3, 2026

Pijukatel force-pushed the claude/actor-resurrection-time-tracking-bixxkl branch from 4b9f3e5 to 8af9ee3 Compare October 20, 2026 07:47

Pijukatel changed the base branch from master to v2 October 20, 2026 07:48

Pijukatel force-pushed the claude/actor-resurrection-time-tracking-bixxkl branch 6 times, most recently from de270b1 to 7ca479e Compare October 21, 2026 06:50

Pijukatel force-pushed the claude/actor-resurrection-time-tracking-bixxkl branch from 7ca479e to fbab2a8 Compare October 21, 2026 06:59

Update comments

fc87368

Pijukatel requested a review from vdusek October 21, 2026 07:04

Pijukatel marked this pull request as ready for review October 21, 2026 07:04

Pijukatel requested a review from apify-factory October 21, 2026 07:09

apify-factory reviewed Oct 21, 2026

View reviewed changes

Pijukatel marked this pull request as draft October 21, 2026 07:38

Pijukatel marked this pull request as ready for review October 21, 2026 07:57

Merge branch 'v2' into claude/actor-resurrection-time-tracking-bixxkl

7ace036

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy

vdusek requested changes Oct 27, 2026

View reviewed changes

Pijukatel and others added 4 commits October 29, 2026 12:00

Apply batched suggestions from code review

41161cd

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

Do not count runtime twice in rare cases when StatisticsState is us…

d5eb284

…ed manually.

Update docs/upgrading/upgrading_to_v2.md

e04085a

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

Update docs/upgrading/upgrading_to_v2.md

63caadb

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

Pijukatel changed the title ~~fix!: do not count downtime towards crawler_runtime after migration or resurrection~~ fix!: exclude downtime from crawler_runtime after a resume Oct 29, 2026

Pijukatel requested a review from vdusek October 29, 2026 12:23

vdusek approved these changes Oct 29, 2026

View reviewed changes

Comment thread src/crawlee/statistics/_statistics.py Resolved

Comment thread tests/unit/_statistics/test_persistence.py Resolved

Comment thread src/crawlee/statistics/_models.py Resolved

Apply batched suggestions from code review

17b87cd

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

Pijukatel merged commit 75b5129 into v2 Oct 30, 2026
37 checks passed

Pijukatel deleted the claude/actor-resurrection-time-tracking-bixxkl branch October 30, 2026 07:51

Conversation

Pijukatel commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Pijukatel commented Sep 3, 2026

Uh oh!

apify-factory left a comment

Choose a reason for hiding this comment

🏭 Factory review — ℹ️ not auto-approved

Uh oh!

Pijukatel commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vdusek commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Pijukatel commented Aug 31, 2026 •

edited

Loading

codecov Bot commented Aug 31, 2026 •

edited

Loading

Pijukatel commented Oct 21, 2026 •

edited

Loading

vdusek commented Oct 27, 2026 •

edited

Loading